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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re Application of: Karnaugh 

Application No.: 09/826690 Examiner Smith, Trad 

Dale Filed: April 5, 2001 Group: 3629 

For: METHOD FOR ADMITTING AN ADMISSIONS APPLICANT INTO AN 
ACADEMIC INSTITUTION 

AFFIDAVIT O P 'STEPHEN P. KLEIN 
I, Stephen P. Klein, being duly sworn, hereby declare: 

1. I hold * B.S. in Psychology from Tufts University, a M.S. in Industrial Psychology, 
with minors in Research Design and Statistics, from Purdue University, and a Fb.D. in Indastrial 
Psychology, with minors in Labor Law and Tests and Measuremciics, from Purdue University. 

2. Since 1974, I have served as a consultant to national and state licensing boards, 
including the National Conference of Bar Examiners, more than 25 state bar boards, the 
American msurote of Certified Public Accountants, the National Board of Professional Teaching 
Standards, the California Commission on Teaching, P" 3 * 0 Board of Medical 
Examiners, and the Society of Actuaries. 

3. Since 1975, 1 have worked at the RAND Corporation and currently hold the title of 
Senior Research Scientist. My wo* requires me to design and direct statistical studies' in me 
fields of education, criminal justice, health, and military manpower. I have served on several 
National Research Council oommirtees and have testified as a measurement and statistical expert 
before legislative committees. 

4. I have expertise in the methods that are employed to obtain adequate inrer-reader 
consistency in grading standards when me answers that were written by a large number of test 
takers to open-ended questions, including essay questions, are evaluated by Afferent readers. 
This expertise includes knowledge of and experience with calibrated grading methods and 
procedures. I also have substantial expertise in the fields of statistics and research design. 
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5, Calibrated grading is a well known terra of an among individuals who participate in 
the administration and grading of seminations, such as essay exams, that require readers to 
evaluate the quality of the teswates' responses. As used in this sense, "calibration" is *e 
process graders go through to help assure that different graders will use consistent standard* 
when evaluating answers. In a well calibrated grading process, a given answer will receive the 
same score regardless of whidi grador evaluated dial answer's quality. It also will receive the 
same score if the same reader evaluated the answer at two or more different points in time. 

6, Calibrate is standard practice on large-scale tests that use multiple readers to 
evaluate test tator responses to written responses, such as on licensing exams &r lawyers and 
teachers, and for the clinical or practical portions of licensing exams, such as far dentists. 
Calibration helps to assure that the ev^^ 

systematic differences among the graders who evaluate the responses. The calibration process is 
designed to reduce the likelihood that a te^ 
wWcbevaluator assessed the quality of that test ts^ 

7, While calibration activities vary somewhat as a function of the specific features of the 
assessment program (such as the number ofevaluators involved and the nature of responses 
being evaluated), they afli^ 

of a common set of performances, and discussion and resolution ofdifferaices in judgment 
among the evahiators. That is why reader calibration, including round robin grading of a 
common batch of answers, is standard practice on large scale essay tests like the new SAT, 
Moreover, professional standards demand that calibration be used in high stakes testing 
pro grams where candidate performance is evaluated by reader*, examiners, raters, or judges. 

8- Calibration is needed regardless of whether an analytic or a holistic grading system is 
used. In an analytic system, the grader assigns points to various parts of a test taker's response, 
such as assigning a separate 

expected to identify and discuss. Inatolistics)^em,ihc 
the answer as a whole. 
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9. Calibration is needed regardless of whether scores are used for making uonn- 
refatmced decisis 

criteriiHw«f«enced decisions (i.e., deciding whether the test taker reached some specified level 
of proficiency). For example, scores on employment and college admissions testa are used to 
makenonn-refereuced decisions whereas scores on licensing exams are used to make criterion- 
referenced decision. The in-car portion of the driver** license test is a good exarcple of lest 
where the score is used to make a criterion-referenced decision. 

10. tosbortjCgllbraticuhc^ to a 

an open-ewtod question is based solely on ibeq^tyof^nsp^crft^tto^chtead^, 
grader, rater, or judge evaluated tot response, 

11. I have no financial interest in the Hfflfoaugh Patent Application (Application 
09/826,690) or any related applications, I am not being paid far my services pertaining to the 
subject matter of this declaration 




State of California 
County of Los Angeles 



j HEREBY CERTIFY that in this day, before me> an officer duly authorized to 
admimster oaths, personally speared Stephen P. Kten, who - _^ ^ ic p g gto n aB y-faiow BtQ po or 
j/^has produced a driver's license as identification, was duly sworn to tell the truth, and 
having done so, says that the foregoing statements are true and correct _ 

IN WITNESS WHEREOF I set my signature and official seal ibis J-* day of 

ft$r2005. 



^^d^ ^AAA^NQTARY PUBLIC 



3k#J*br r^Print Name \^g^ ^KnTl& 
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Notary Pubfic - C«romja | 
Los AngeiB* CounTy^f 



